Improve handling of constraints on requirements with extras #12095

sanderr · 2023-06-22T14:54:32Z

Description

These changes improve the handling of constraints on requirements with extras (#11924). It includes the following changes:

Constraints are now all taken into account for the base candidate. To this end hello[world]~=1.0 now produces the implicit hello~=1.0 requirement. This allows us to provide resolvelib with a more representative candidate list for the base. The requirements with an extra then serve only to require the extra for the selected package. In cases where multiple constraints occur for the same package but with different extras (see examples below) this greatly reduces the number of candidates that need to be visited, and therefore downloaded.
Minor bugfix in comes_from tracking for requirements with extras: pip._internal.resolution.resolvelib.factory keeps a cache for link candidates, which would result in the very first requirement for a project being considered the origin of any candidates for it, regardless of the extras on the candidate or requirement. This bug exists on the main branch but with the changes I made for 1 it triggered consistently so I had to fix it in order to keep install reports stable. See examples below.
Minor bugfix in install report generation where only the very first requirement for a project would be considered for requested_extras in the report. Similar to 2, this bug can be triggered on main but it became consistent with the changes I made for 1. See examples below.

closes #11924
closes #12210

Status

This pull request still requires some minor refinement and cleanup, more specifically I still need to add tests, run style checks, fix compatibility with older Python versions (nox -s test-3.11 currently succeeds on my system) and add a news fragment. Before I get to that I thought it would be useful to already open it up as a draft pull request to get an initial conceptual review. If my approach has some fundamental problems I think this is the best time to address it.

Ready for review

Examples

#11924 contains a few examples of the issue. Each of those has been greatly improved by these changes. For consistency I'll limit myself to more minimal examples here:

pip download Paste Paste[openid]==1.7.5.1 works as expected on the latest stable pip but pip download Paste==1.7.5.1 Paste[openid] downloads 35 versions of Paste, no matter which order the two constraints appear in. With the changes in this pull request only the requested version is downloaded. Similar observations can be made for Paste[subprocess] ==1.7.5.1 Paste[openid] and 'Paste[subprocess]==1.7.5.1' 'Paste[openid]>=1.8' (which is trivially rejected with these changes).
With the latest stable pip download Paste Paste[openid] would log a line Collecting setuptools (from Paste). If you swap the order of the requirements it would get it right: Collecting setuptools (from Paste[openid]). With the changes in this pull request it will always get it right.
With the latest stable pip install --dry-run --report Paste==1.7.5.1 'Paste[openid]==1.7.5.1' would not include any requested_extras in the report. Swapping the requirements would report the requested extras correctly. Similarly for pip install --dry-run --report myreport.txt Paste[openid] Paste[subprocess] only openid would be in the report. With the changes in this pull request all requested extras will be present.

src/pip/_internal/resolution/resolvelib/requirements.py

uranusjr · 2023-07-06T09:16:36Z

src/pip/_internal/req/constructors.py

+    req = Requirement(str(ireq.req))
+    req.extras = {}


Instead of trying to modify a Requirement in-place (which IIRC is not documented to be an officially supported operation), it may be better to strip the extras in the requirement string instead. This should not be too dificult with some regex since both the project and extra names are quite predictable.

(I’m not committed to this though and wouldn’t object if packaging maintainers are OK with this cc @pradyunsg)

I made the change though I don't like the fact that it feels a bit more brittle. On the other hand I do agree with your sentiment that I shouldn't something that isn't part of packaging.Requirement's contract. @pradyunsg do you have an opinion on this (faa3289)?

I'd be happy if packaging made it a supported operation, but the existing docs state that the Requirement class is for parsing the string form of a requirement, which to me doesn't imply the result is mutable. You could read it as saying that it parses the string into a structured form, which you are allowed to them mutate if you want, but I don't think that automatically follows.

So as things stand, I agree with @uranusjr.

As @pradyunsg is a packaging maintainer, if he says it's OK, then I'm fine with that (although I'd ask that the docs be changed to explicitly state what's supported).

Yea, don't modify this in place. It doesn't break things as implemented right now but making this a frozen dataclass is something I want to do in the future.

Thanks for your input. A frozen dataclass would be nice, that would then allow for dataclasses.replace, which I'd always prefer over in-place modifications.

news/12095.bugfix.rst

uranusjr · 2023-07-26T06:17:33Z

src/pip/_internal/req/constructors.py

+    match: Optional[re.Match[str]] = re.fullmatch(
+        r"([^;\[<>~=]+)(\[[^\]]*\])?(.*)", str(req)
+    )


Do we need the regex to be this complicated? I thought the extra is pretty easy to detect, isn’t all needed is to scan for \w and split the first non-match?

You make a good point, if I interpret PEP508 correctly extras (if present) will always come right after the name. I don't think \w suffices because dots and dashes are allowed as well, but even with that modification to your proposal it should be simpler than what I used here.

Yes, the extras come immediately after, so we can scan for the name, skip any spaces if needed, remove existing extras if needed, and add the extras right in.

Applied suggestion in 314d7c1

sanderr · 2023-10-04T14:07:43Z

@uranusjr @pradyunsg is there anything still blocking this from being merged? I rerequested review from @uranusjr, but I'm honestly not sure what your typical procedure for reviews is, I couldn't find anything about that in the contributing guidelines. If it's just that you've been too busy to give this your attention I don't mind being patient, I just want to make sure it's still on the radar and not blocked on something from my side.

pradyunsg · 2023-10-04T19:30:44Z

Thanks for checking! It's mostly availability, if I understand correctly.

I've added this to the milestone for the upcoming release. If @uranusjr isn't able to take a look at this by then, due to lack of availability, let's land this (assuming the RM doesn't have concerns around this change).

uranusjr

I really don’t like how Black formats some of the code but that’s not your fault and there’s not much we can do 😮‍💨

uranusjr · 2023-10-05T07:29:21Z

I’ll just merge since reading the above discussion I seem to be the only blocker anyway.

…k/test/generated-code (#4296) Bumps [pip](https://github.com/pypa/pip) from 23.2.1 to 23.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>23.3 (2023-10-15)</h1> <h2>Process</h2> <ul> <li>Added reference to <code>vulnerability reporting guidelines <https://www.python.org/dev/security/></code>_ to pip's security policy.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (<code>[#12175](pypa/pip#12175) <https://github.com/pypa/pip/issues/12175></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Improve extras resolution for multiple constraints on same base package. (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Improve use of datastructures to make candidate selection 1.6x faster. (<code>[#12204](pypa/pip#12204) <https://github.com/pypa/pip/issues/12204></code>_)</li> <li>Allow <code>pip install --dry-run</code> to use platform and ABI overriding options. (<code>[#12215](pypa/pip#12215) <https://github.com/pypa/pip/issues/12215></code>_)</li> <li>Add <code>is_yanked</code> boolean entry to the installation report (<code>--report</code>) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:<code>592</code>. (<code>[#12224](pypa/pip#12224) <https://github.com/pypa/pip/issues/12224></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Ignore errors in temporary directory cleanup (show a warning instead). (<code>[#11394](pypa/pip#11394) <https://github.com/pypa/pip/issues/11394></code>_)</li> <li>Normalize extras according to :pep:<code>685</code> from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this <em>does not</em> solve cases where the package itself contains unnormalized extra values in the metadata. (<code>[#11649](pypa/pip#11649) <https://github.com/pypa/pip/issues/11649></code>_)</li> <li>Prevent downloading sdists twice when :pep:<code>658</code> metadata is present. (<code>[#11847](pypa/pip#11847) <https://github.com/pypa/pip/issues/11847></code>_)</li> <li>Include all requested extras in the install report (<code>--report</code>). (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Removed uses of <code>datetime.datetime.utcnow</code> from non-vendored code. (<code>[#12005](pypa/pip#12005) <https://github.com/pypa/pip/issues/12005></code>_)</li> <li>Consistently report whether a dependency comes from an extra. (<code>[#12095](pypa/pip#12095) <https://github.com/pypa/pip/issues/12095></code>_)</li> <li>Fix completion script for zsh (<code>[#12166](pypa/pip#12166) <https://github.com/pypa/pip/issues/12166></code>_)</li> <li>Fix improper handling of the new onexc argument of <code>shutil.rmtree()</code> in Python 3.12. (<code>[#12187](pypa/pip#12187) <https://github.com/pypa/pip/issues/12187></code>_)</li> <li>Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (<code>[#12225](pypa/pip#12225) <https://github.com/pypa/pip/issues/12225></code>_)</li> <li>Fix crash when the git version number contains something else than digits and dots. (<code>[#12280](pypa/pip#12280) <https://github.com/pypa/pip/issues/12280></code>_)</li> <li>Use <code>-r=...</code> instead of <code>-r ...</code> to specify references with Mercurial. (<code>[#12306](pypa/pip#12306) <https://github.com/pypa/pip/issues/12306></code>_)</li> <li>Redact password from URLs in some additional places. (<code>[#12350](pypa/pip#12350) <https://github.com/pypa/pip/issues/12350></code>_)</li> <li>pip uses less memory when caching large packages. As a result, there is a new on-disk cache format stored in a new directory ($PIP_CACHE_DIR/http-v2). (<code>[#2984](pypa/pip#2984) <https://github.com/pypa/pip/issues/2984></code>_)</li> </ul> <h2>Vendored Libraries</h2> <ul> <li>Upgrade certifi to 2023.7.22</li> <li>Add truststore 0.8.0</li> <li>Upgrade urllib3 to 1.26.17</li> </ul> <p>Improved Documentation</p> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/e3dc91dad93a020b3034a87ebe59027f63370fe8"><code>e3dc91d</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/3e85558b10722598fb3353126e2f19979f7cf7dd"><code>3e85558</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/8d0278771c7325b04f02cb073c8ef02827cbeb93"><code>8d02787</code></a> Reclassify news fragment</li> <li><a href="https://github.com/pypa/pip/commit/f6ecf406c3929b3127ddb480ef4350542d102338"><code>f6ecf40</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12350">#12350</a> from sbidoul/readact-collecting-url</li> <li><a href="https://github.com/pypa/pip/commit/306086513bd1a6500126057492ee8b0f9a2e79dd"><code>3060865</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12335">#12335</a> from edmorley/patch-1</li> <li><a href="https://github.com/pypa/pip/commit/8f0ed32413daa411a728b50cd7776b9c02b010d5"><code>8f0ed32</code></a> Redact URLs in Collecting... logs</li> <li><a href="https://github.com/pypa/pip/commit/d1659b87e46abd0a2dcc74f2160dd52e6190e13b"><code>d1659b8</code></a> Correct issue number for NEWS entry added by <a href="https://github.com/pypa/pip/issues/12197">#12197</a></li> <li><a href="https://github.com/pypa/pip/commit/2333ef3b53a71fb7acc9e76d6ff90409576b2250"><code>2333ef3</code></a> Upgrade urllib3 to 1.26.17 (<a href="https://github.com/pypa/pip/issues/12343">#12343</a>)</li> <li><a href="https://github.com/pypa/pip/commit/496b268c1b9ce3466c08eb4819e5460a943d1793"><code>496b268</code></a> Update "Running Tests" documentation (<a href="https://github.com/pypa/pip/issues/12334">#12334</a>)</li> <li><a href="https://github.com/pypa/pip/commit/d1f0981cb2af3c72ff871b54a8a98581ccb2890a"><code>d1f0981</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12331">#12331</a> from sbidoul/update-egg-deprecation-message</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/23.2.1...23.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=23.2.1&new-version=23.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

…s/@jsii/python-runtime (#4295) Updates the requirements on [pip](https://github.com/pypa/pip) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>23.3 (2023-10-15)</h1> <h2>Process</h2> <ul> <li>Added reference to <code>vulnerability reporting guidelines <https://www.python.org/dev/security/></code>_ to pip's security policy.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (<code>[#12175](pypa/pip#12175) <https://github.com/pypa/pip/issues/12175></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Improve extras resolution for multiple constraints on same base package. (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Improve use of datastructures to make candidate selection 1.6x faster. (<code>[#12204](pypa/pip#12204) <https://github.com/pypa/pip/issues/12204></code>_)</li> <li>Allow <code>pip install --dry-run</code> to use platform and ABI overriding options. (<code>[#12215](pypa/pip#12215) <https://github.com/pypa/pip/issues/12215></code>_)</li> <li>Add <code>is_yanked</code> boolean entry to the installation report (<code>--report</code>) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:<code>592</code>. (<code>[#12224](pypa/pip#12224) <https://github.com/pypa/pip/issues/12224></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Ignore errors in temporary directory cleanup (show a warning instead). (<code>[#11394](pypa/pip#11394) <https://github.com/pypa/pip/issues/11394></code>_)</li> <li>Normalize extras according to :pep:<code>685</code> from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this <em>does not</em> solve cases where the package itself contains unnormalized extra values in the metadata. (<code>[#11649](pypa/pip#11649) <https://github.com/pypa/pip/issues/11649></code>_)</li> <li>Prevent downloading sdists twice when :pep:<code>658</code> metadata is present. (<code>[#11847](pypa/pip#11847) <https://github.com/pypa/pip/issues/11847></code>_)</li> <li>Include all requested extras in the install report (<code>--report</code>). (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Removed uses of <code>datetime.datetime.utcnow</code> from non-vendored code. (<code>[#12005](pypa/pip#12005) <https://github.com/pypa/pip/issues/12005></code>_)</li> <li>Consistently report whether a dependency comes from an extra. (<code>[#12095](pypa/pip#12095) <https://github.com/pypa/pip/issues/12095></code>_)</li> <li>Fix completion script for zsh (<code>[#12166](pypa/pip#12166) <https://github.com/pypa/pip/issues/12166></code>_)</li> <li>Fix improper handling of the new onexc argument of <code>shutil.rmtree()</code> in Python 3.12. (<code>[#12187](pypa/pip#12187) <https://github.com/pypa/pip/issues/12187></code>_)</li> <li>Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (<code>[#12225](pypa/pip#12225) <https://github.com/pypa/pip/issues/12225></code>_)</li> <li>Fix crash when the git version number contains something else than digits and dots. (<code>[#12280](pypa/pip#12280) <https://github.com/pypa/pip/issues/12280></code>_)</li> <li>Use <code>-r=...</code> instead of <code>-r ...</code> to specify references with Mercurial. (<code>[#12306](pypa/pip#12306) <https://github.com/pypa/pip/issues/12306></code>_)</li> <li>Redact password from URLs in some additional places. (<code>[#12350](pypa/pip#12350) <https://github.com/pypa/pip/issues/12350></code>_)</li> <li>pip uses less memory when caching large packages. As a result, there is a new on-disk cache format stored in a new directory ($PIP_CACHE_DIR/http-v2). (<code>[#2984](pypa/pip#2984) <https://github.com/pypa/pip/issues/2984></code>_)</li> </ul> <h2>Vendored Libraries</h2> <ul> <li>Upgrade certifi to 2023.7.22</li> <li>Add truststore 0.8.0</li> <li>Upgrade urllib3 to 1.26.17</li> </ul> <p>Improved Documentation</p> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/e3dc91dad93a020b3034a87ebe59027f63370fe8"><code>e3dc91d</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/3e85558b10722598fb3353126e2f19979f7cf7dd"><code>3e85558</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/8d0278771c7325b04f02cb073c8ef02827cbeb93"><code>8d02787</code></a> Reclassify news fragment</li> <li><a href="https://github.com/pypa/pip/commit/f6ecf406c3929b3127ddb480ef4350542d102338"><code>f6ecf40</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12350">#12350</a> from sbidoul/readact-collecting-url</li> <li><a href="https://github.com/pypa/pip/commit/306086513bd1a6500126057492ee8b0f9a2e79dd"><code>3060865</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12335">#12335</a> from edmorley/patch-1</li> <li><a href="https://github.com/pypa/pip/commit/8f0ed32413daa411a728b50cd7776b9c02b010d5"><code>8f0ed32</code></a> Redact URLs in Collecting... logs</li> <li><a href="https://github.com/pypa/pip/commit/d1659b87e46abd0a2dcc74f2160dd52e6190e13b"><code>d1659b8</code></a> Correct issue number for NEWS entry added by <a href="https://github.com/pypa/pip/issues/12197">#12197</a></li> <li><a href="https://github.com/pypa/pip/commit/2333ef3b53a71fb7acc9e76d6ff90409576b2250"><code>2333ef3</code></a> Upgrade urllib3 to 1.26.17 (<a href="https://github.com/pypa/pip/issues/12343">#12343</a>)</li> <li><a href="https://github.com/pypa/pip/commit/496b268c1b9ce3466c08eb4819e5460a943d1793"><code>496b268</code></a> Update "Running Tests" documentation (<a href="https://github.com/pypa/pip/issues/12334">#12334</a>)</li> <li><a href="https://github.com/pypa/pip/commit/d1f0981cb2af3c72ff871b54a8a98581ccb2890a"><code>d1f0981</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12331">#12331</a> from sbidoul/update-egg-deprecation-message</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/23.2...23.3">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

Bumps [pip](https://github.com/pypa/pip) from 23.2.1 to 23.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>23.3 (2023-10-15)</h1> <h2>Process</h2> <ul> <li>Added reference to <code>vulnerability reporting guidelines <https://www.python.org/dev/security/></code>_ to pip's security policy.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (<code>[#12175](pypa/pip#12175) <https://github.com/pypa/pip/issues/12175></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Improve extras resolution for multiple constraints on same base package. (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Improve use of datastructures to make candidate selection 1.6x faster. (<code>[#12204](pypa/pip#12204) <https://github.com/pypa/pip/issues/12204></code>_)</li> <li>Allow <code>pip install --dry-run</code> to use platform and ABI overriding options. (<code>[#12215](pypa/pip#12215) <https://github.com/pypa/pip/issues/12215></code>_)</li> <li>Add <code>is_yanked</code> boolean entry to the installation report (<code>--report</code>) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:<code>592</code>. (<code>[#12224](pypa/pip#12224) <https://github.com/pypa/pip/issues/12224></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Ignore errors in temporary directory cleanup (show a warning instead). (<code>[#11394](pypa/pip#11394) <https://github.com/pypa/pip/issues/11394></code>_)</li> <li>Normalize extras according to :pep:<code>685</code> from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this <em>does not</em> solve cases where the package itself contains unnormalized extra values in the metadata. (<code>[#11649](pypa/pip#11649) <https://github.com/pypa/pip/issues/11649></code>_)</li> <li>Prevent downloading sdists twice when :pep:<code>658</code> metadata is present. (<code>[#11847](pypa/pip#11847) <https://github.com/pypa/pip/issues/11847></code>_)</li> <li>Include all requested extras in the install report (<code>--report</code>). (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Removed uses of <code>datetime.datetime.utcnow</code> from non-vendored code. (<code>[#12005](pypa/pip#12005) <https://github.com/pypa/pip/issues/12005></code>_)</li> <li>Consistently report whether a dependency comes from an extra. (<code>[#12095](pypa/pip#12095) <https://github.com/pypa/pip/issues/12095></code>_)</li> <li>Fix completion script for zsh (<code>[#12166](pypa/pip#12166) <https://github.com/pypa/pip/issues/12166></code>_)</li> <li>Fix improper handling of the new onexc argument of <code>shutil.rmtree()</code> in Python 3.12. (<code>[#12187](pypa/pip#12187) <https://github.com/pypa/pip/issues/12187></code>_)</li> <li>Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (<code>[#12225](pypa/pip#12225) <https://github.com/pypa/pip/issues/12225></code>_)</li> <li>Fix crash when the git version number contains something else than digits and dots. (<code>[#12280](pypa/pip#12280) <https://github.com/pypa/pip/issues/12280></code>_)</li> <li>Use <code>-r=...</code> instead of <code>-r ...</code> to specify references with Mercurial. (<code>[#12306](pypa/pip#12306) <https://github.com/pypa/pip/issues/12306></code>_)</li> <li>Redact password from URLs in some additional places. (<code>[#12350](pypa/pip#12350) <https://github.com/pypa/pip/issues/12350></code>_)</li> <li>pip uses less memory when caching large packages. As a result, there is a new on-disk cache format stored in a new directory ($PIP_CACHE_DIR/http-v2). (<code>[#2984](pypa/pip#2984) <https://github.com/pypa/pip/issues/2984></code>_)</li> </ul> <h2>Vendored Libraries</h2> <ul> <li>Upgrade certifi to 2023.7.22</li> <li>Add truststore 0.8.0</li> <li>Upgrade urllib3 to 1.26.17</li> </ul> <p>Improved Documentation</p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/e3dc91dad93a020b3034a87ebe59027f63370fe8"><code>e3dc91d</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/3e85558b10722598fb3353126e2f19979f7cf7dd"><code>3e85558</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/8d0278771c7325b04f02cb073c8ef02827cbeb93"><code>8d02787</code></a> Reclassify news fragment</li> <li><a href="https://github.com/pypa/pip/commit/f6ecf406c3929b3127ddb480ef4350542d102338"><code>f6ecf40</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12350">#12350</a> from sbidoul/readact-collecting-url</li> <li><a href="https://github.com/pypa/pip/commit/306086513bd1a6500126057492ee8b0f9a2e79dd"><code>3060865</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12335">#12335</a> from edmorley/patch-1</li> <li><a href="https://github.com/pypa/pip/commit/8f0ed32413daa411a728b50cd7776b9c02b010d5"><code>8f0ed32</code></a> Redact URLs in Collecting... logs</li> <li><a href="https://github.com/pypa/pip/commit/d1659b87e46abd0a2dcc74f2160dd52e6190e13b"><code>d1659b8</code></a> Correct issue number for NEWS entry added by <a href="https://github.com/pypa/pip/issues/12197">#12197</a></li> <li><a href="https://github.com/pypa/pip/commit/2333ef3b53a71fb7acc9e76d6ff90409576b2250"><code>2333ef3</code></a> Upgrade urllib3 to 1.26.17 (<a href="https://github.com/pypa/pip/issues/12343">#12343</a>)</li> <li><a href="https://github.com/pypa/pip/commit/496b268c1b9ce3466c08eb4819e5460a943d1793"><code>496b268</code></a> Update "Running Tests" documentation (<a href="https://github.com/pypa/pip/issues/12334">#12334</a>)</li> <li><a href="https://github.com/pypa/pip/commit/d1f0981cb2af3c72ff871b54a8a98581ccb2890a"><code>d1f0981</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12331">#12331</a> from sbidoul/update-egg-deprecation-message</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/23.2.1...23.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=23.2.1&new-version=23.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

Bumps [pip](https://github.com/pypa/pip) from 23.2.1 to 23.3.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>23.3.1 (2023-10-21)</h1> <h2>Bug Fixes</h2> <ul> <li>Handle a timezone indicator of Z when parsing dates in the self check. (<code>[#12338](pypa/pip#12338) <https://github.com/pypa/pip/issues/12338></code>_)</li> <li>Fix bug where installing the same package at the same time with multiple pip processes could fail. (<code>[#12361](pypa/pip#12361) <https://github.com/pypa/pip/issues/12361></code>_)</li> </ul> <h1>23.3 (2023-10-15)</h1> <h2>Process</h2> <ul> <li>Added reference to <code>vulnerability reporting guidelines <https://www.python.org/dev/security/></code>_ to pip's security policy.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (<code>[#12175](pypa/pip#12175) <https://github.com/pypa/pip/issues/12175></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Improve extras resolution for multiple constraints on same base package. (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Improve use of datastructures to make candidate selection 1.6x faster. (<code>[#12204](pypa/pip#12204) <https://github.com/pypa/pip/issues/12204></code>_)</li> <li>Allow <code>pip install --dry-run</code> to use platform and ABI overriding options. (<code>[#12215](pypa/pip#12215) <https://github.com/pypa/pip/issues/12215></code>_)</li> <li>Add <code>is_yanked</code> boolean entry to the installation report (<code>--report</code>) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:<code>592</code>. (<code>[#12224](pypa/pip#12224) <https://github.com/pypa/pip/issues/12224></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Ignore errors in temporary directory cleanup (show a warning instead). (<code>[#11394](pypa/pip#11394) <https://github.com/pypa/pip/issues/11394></code>_)</li> <li>Normalize extras according to :pep:<code>685</code> from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this <em>does not</em> solve cases where the package itself contains unnormalized extra values in the metadata. (<code>[#11649](pypa/pip#11649) <https://github.com/pypa/pip/issues/11649></code>_)</li> <li>Prevent downloading sdists twice when :pep:<code>658</code> metadata is present. (<code>[#11847](pypa/pip#11847) <https://github.com/pypa/pip/issues/11847></code>_)</li> <li>Include all requested extras in the install report (<code>--report</code>). (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Removed uses of <code>datetime.datetime.utcnow</code> from non-vendored code. (<code>[#12005](pypa/pip#12005) <https://github.com/pypa/pip/issues/12005></code>_)</li> <li>Consistently report whether a dependency comes from an extra. (<code>[#12095](pypa/pip#12095) <https://github.com/pypa/pip/issues/12095></code>_)</li> <li>Fix completion script for zsh (<code>[#12166](pypa/pip#12166) <https://github.com/pypa/pip/issues/12166></code>_)</li> <li>Fix improper handling of the new onexc argument of <code>shutil.rmtree()</code> in Python 3.12. (<code>[#12187](pypa/pip#12187) <https://github.com/pypa/pip/issues/12187></code>_)</li> <li>Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (<code>[#12225](pypa/pip#12225) <https://github.com/pypa/pip/issues/12225></code>_)</li> <li>Fix crash when the git version number contains something else than digits and dots. (<code>[#12280](pypa/pip#12280) <https://github.com/pypa/pip/issues/12280></code>_)</li> <li>Use <code>-r=...</code> instead of <code>-r ...</code> to specify references with Mercurial. (<code>[#12306](pypa/pip#12306) <https://github.com/pypa/pip/issues/12306></code>_)</li> <li>Redact password from URLs in some additional places. (<code>[#12350](pypa/pip#12350) <https://github.com/pypa/pip/issues/12350></code>_)</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/5364f26f9631dc07ed1bdfc88e1bec1bead2bce3"><code>5364f26</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/5e7cc16c3b4442055a4a9892e9231758b6714e28"><code>5e7cc16</code></a> Fix parallel pip cache downloads causing crash (<a href="https://github.com/pypa/pip/issues/12364">#12364</a>)</li> <li><a href="https://github.com/pypa/pip/commit/8a0f77c171d60344e6a3bf6e95ad5740c21575fd"><code>8a0f77c</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12355">#12355</a> from sbidoul/build-using-build</li> <li><a href="https://github.com/pypa/pip/commit/f3620cdb5be06cee223a3606a1525ee45372085b"><code>f3620cd</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12363">#12363</a> from pfmoore/safe_isoformat</li> <li><a href="https://github.com/pypa/pip/commit/fb06d12d5a32581ae531fc26143c14ac6c8ea8fe"><code>fb06d12</code></a> Handle ISO formats with a trailing Z</li> <li><a href="https://github.com/pypa/pip/commit/9f213bf69ac32c60c84055261c862ff169389e43"><code>9f213bf</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12356">#12356</a> from sbidoul/clarify-changelog</li> <li><a href="https://github.com/pypa/pip/commit/a982c7bc3550afb27a3a792d84fe91bf7c3254ca"><code>a982c7b</code></a> Add a few PEP links in the changelog</li> <li><a href="https://github.com/pypa/pip/commit/e1e227d7d6b5ae04ae3a2104bf8185622201f5f6"><code>e1e227d</code></a> Clarify changelog</li> <li><a href="https://github.com/pypa/pip/commit/9b0abc8c40459dd16a9c1205e15f6d3363bf202e"><code>9b0abc8</code></a> Build using <code>build</code></li> <li><a href="https://github.com/pypa/pip/commit/9d4be7802f45790bdb994f943c8d8731927cf25c"><code>9d4be78</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12353">#12353</a> from sbidoul/release/23.3</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/23.2.1...23.3.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=23.2.1&new-version=23.3.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

AndydeCleyre · 2023-10-26T03:43:47Z

Hello! This introduced a change in pip-tools behavior (jazzband/pip-tools#2003), as a bunch of non-guaranteed pip internals are still used for now.

Before addressing it in pip-tools, I wanted to check that the change here was intentional. It seems from point 1 of the description that it may indeed be.

As of 5f8f40e, we're getting for example coverage listed as a dependency of coverage[toml] in some cases, demonstrated with the first example here, where it's coming in as a dependency of pytest-cov:

$ <<<'pytest-cov' pip-compile --annotation-style line - -o - 2>/dev/null | grep coverage
coverage[toml]==7.3.2     # via coverage, pytest-cov

$ <<<'coverage[toml]' pip-compile --annotation-style line - -o - 2>/dev/null | grep coverage
coverage[toml]==7.3.2     # via -r -

$ <<<'pytest-cov'$'\n''coverage[toml]' pip-compile --annotation-style line - -o - 2>/dev/null | grep coverage
coverage[toml]==7.3.2     # via -r -, pytest-cov

I'm guessing that this is just a pip-tools problem where we need to account for the change, but mention it here in case it indicates a surprise effect.

sanderr · 2023-10-26T11:41:42Z

@AndydeCleyre I just had a look at this and I am inclined to agree that this is an intentional internal change. As far as I can tell the unexpected "via coverage" comes from the InstallRequirement "coverage[toml]==7.3.2 from https://<...>.whl (from coverage[toml]>=5.2.1->pytest-cov->-r -)". Before this pull request this would be just "... (from pytest-cov->-r -)".

As you already inferred, the change in behavior is due to point 1 in the description: coverage[toml]>=5.2.1 now results in two distinct requirements: coverage>=5.2.1 and coverage[toml]. Semantically, I feel that it is appropriate that these requirements point to coverage[toml]>=5.2.1 as their origin (comes_from). It is only because pip-compile trims some of the constraint metadata (I don't mean to say that it shouldn't) that it comes across as strange.

With the following rough patch I get the old behavior back. I'm not sure if the name check is a good/stable metric, but it may be of use in finding a proper solution.

diff --git a/piptools/writer.py b/piptools/writer.py
index 1ae341d..92a2a33 100644
--- a/piptools/writer.py
+++ b/piptools/writer.py
@@ -295,7 +295,7 @@ class OutputWriter:
                 if src_ireq.comes_from
             }

-        if ireq.comes_from:
+        if ireq.comes_from and (isinstance(ireq.comes_from, str) or ireq.comes_from.name != ireq.name):
             required_by.add(_comes_from_as_string(ireq.comes_from))

         required_by |= set(getattr(ireq, "_required_by", set()))

To be clear: I'm not a pip maintainer, so this comment is not authoritative and represents nothing more than my view on the situation. Let me know if you have further questions or if you disagree with any of the above.

PS: sorry about the commit history you had to wade through. I'm used to pull request commits being squashed before merging so I don't always pay too much attention to make clear commits.

…ements-on-extras" This reverts commit b551c09, reversing changes made to dfaac0a.

Bumps [pip](https://github.com/pypa/pip) from 23.2.1 to 23.3.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>23.3.1 (2023-10-21)</h1> <h2>Bug Fixes</h2> <ul> <li>Handle a timezone indicator of Z when parsing dates in the self check. (<code>[#12338](pypa/pip#12338) <https://github.com/pypa/pip/issues/12338></code>_)</li> <li>Fix bug where installing the same package at the same time with multiple pip processes could fail. (<code>[#12361](pypa/pip#12361) <https://github.com/pypa/pip/issues/12361></code>_)</li> </ul> <h1>23.3 (2023-10-15)</h1> <h2>Process</h2> <ul> <li>Added reference to <code>vulnerability reporting guidelines <https://www.python.org/dev/security/></code>_ to pip's security policy.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (<code>[#12175](pypa/pip#12175) <https://github.com/pypa/pip/issues/12175></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Improve extras resolution for multiple constraints on same base package. (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Improve use of datastructures to make candidate selection 1.6x faster. (<code>[#12204](pypa/pip#12204) <https://github.com/pypa/pip/issues/12204></code>_)</li> <li>Allow <code>pip install --dry-run</code> to use platform and ABI overriding options. (<code>[#12215](pypa/pip#12215) <https://github.com/pypa/pip/issues/12215></code>_)</li> <li>Add <code>is_yanked</code> boolean entry to the installation report (<code>--report</code>) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:<code>592</code>. (<code>[#12224](pypa/pip#12224) <https://github.com/pypa/pip/issues/12224></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Ignore errors in temporary directory cleanup (show a warning instead). (<code>[#11394](pypa/pip#11394) <https://github.com/pypa/pip/issues/11394></code>_)</li> <li>Normalize extras according to :pep:<code>685</code> from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this <em>does not</em> solve cases where the package itself contains unnormalized extra values in the metadata. (<code>[#11649](pypa/pip#11649) <https://github.com/pypa/pip/issues/11649></code>_)</li> <li>Prevent downloading sdists twice when :pep:<code>658</code> metadata is present. (<code>[#11847](pypa/pip#11847) <https://github.com/pypa/pip/issues/11847></code>_)</li> <li>Include all requested extras in the install report (<code>--report</code>). (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Removed uses of <code>datetime.datetime.utcnow</code> from non-vendored code. (<code>[#12005](pypa/pip#12005) <https://github.com/pypa/pip/issues/12005></code>_)</li> <li>Consistently report whether a dependency comes from an extra. (<code>[#12095](pypa/pip#12095) <https://github.com/pypa/pip/issues/12095></code>_)</li> <li>Fix completion script for zsh (<code>[#12166](pypa/pip#12166) <https://github.com/pypa/pip/issues/12166></code>_)</li> <li>Fix improper handling of the new onexc argument of <code>shutil.rmtree()</code> in Python 3.12. (<code>[#12187](pypa/pip#12187) <https://github.com/pypa/pip/issues/12187></code>_)</li> <li>Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (<code>[#12225](pypa/pip#12225) <https://github.com/pypa/pip/issues/12225></code>_)</li> <li>Fix crash when the git version number contains something else than digits and dots. (<code>[#12280](pypa/pip#12280) <https://github.com/pypa/pip/issues/12280></code>_)</li> <li>Use <code>-r=...</code> instead of <code>-r ...</code> to specify references with Mercurial. (<code>[#12306](pypa/pip#12306) <https://github.com/pypa/pip/issues/12306></code>_)</li> <li>Redact password from URLs in some additional places. (<code>[#12350](pypa/pip#12350) <https://github.com/pypa/pip/issues/12350></code>_)</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/5364f26f9631dc07ed1bdfc88e1bec1bead2bce3"><code>5364f26</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/5e7cc16c3b4442055a4a9892e9231758b6714e28"><code>5e7cc16</code></a> Fix parallel pip cache downloads causing crash (<a href="https://github.com/pypa/pip/issues/12364">#12364</a>)</li> <li><a href="https://github.com/pypa/pip/commit/8a0f77c171d60344e6a3bf6e95ad5740c21575fd"><code>8a0f77c</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12355">#12355</a> from sbidoul/build-using-build</li> <li><a href="https://github.com/pypa/pip/commit/f3620cdb5be06cee223a3606a1525ee45372085b"><code>f3620cd</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12363">#12363</a> from pfmoore/safe_isoformat</li> <li><a href="https://github.com/pypa/pip/commit/fb06d12d5a32581ae531fc26143c14ac6c8ea8fe"><code>fb06d12</code></a> Handle ISO formats with a trailing Z</li> <li><a href="https://github.com/pypa/pip/commit/9f213bf69ac32c60c84055261c862ff169389e43"><code>9f213bf</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12356">#12356</a> from sbidoul/clarify-changelog</li> <li><a href="https://github.com/pypa/pip/commit/a982c7bc3550afb27a3a792d84fe91bf7c3254ca"><code>a982c7b</code></a> Add a few PEP links in the changelog</li> <li><a href="https://github.com/pypa/pip/commit/e1e227d7d6b5ae04ae3a2104bf8185622201f5f6"><code>e1e227d</code></a> Clarify changelog</li> <li><a href="https://github.com/pypa/pip/commit/9b0abc8c40459dd16a9c1205e15f6d3363bf202e"><code>9b0abc8</code></a> Build using <code>build</code></li> <li><a href="https://github.com/pypa/pip/commit/9d4be7802f45790bdb994f943c8d8731927cf25c"><code>9d4be78</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12353">#12353</a> from sbidoul/release/23.3</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/23.2.1...23.3.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=23.2.1&new-version=23.3.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pip](https://github.com/pypa/pip) from 22.3.1 to 23.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>23.3 (2023-10-15)</h1> <h2>Process</h2> <ul> <li>Added reference to <code>vulnerability reporting guidelines <https://www.python.org/dev/security/></code>_ to pip's security policy.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (<code>[#12175](pypa/pip#12175) <https://github.com/pypa/pip/issues/12175></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Improve extras resolution for multiple constraints on same base package. (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Improve use of datastructures to make candidate selection 1.6x faster. (<code>[#12204](pypa/pip#12204) <https://github.com/pypa/pip/issues/12204></code>_)</li> <li>Allow <code>pip install --dry-run</code> to use platform and ABI overriding options. (<code>[#12215](pypa/pip#12215) <https://github.com/pypa/pip/issues/12215></code>_)</li> <li>Add <code>is_yanked</code> boolean entry to the installation report (<code>--report</code>) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:<code>592</code>. (<code>[#12224](pypa/pip#12224) <https://github.com/pypa/pip/issues/12224></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Ignore errors in temporary directory cleanup (show a warning instead). (<code>[#11394](pypa/pip#11394) <https://github.com/pypa/pip/issues/11394></code>_)</li> <li>Normalize extras according to :pep:<code>685</code> from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this <em>does not</em> solve cases where the package itself contains unnormalized extra values in the metadata. (<code>[#11649](pypa/pip#11649) <https://github.com/pypa/pip/issues/11649></code>_)</li> <li>Prevent downloading sdists twice when :pep:<code>658</code> metadata is present. (<code>[#11847](pypa/pip#11847) <https://github.com/pypa/pip/issues/11847></code>_)</li> <li>Include all requested extras in the install report (<code>--report</code>). (<code>[#11924](pypa/pip#11924) <https://github.com/pypa/pip/issues/11924></code>_)</li> <li>Removed uses of <code>datetime.datetime.utcnow</code> from non-vendored code. (<code>[#12005](pypa/pip#12005) <https://github.com/pypa/pip/issues/12005></code>_)</li> <li>Consistently report whether a dependency comes from an extra. (<code>[#12095](pypa/pip#12095) <https://github.com/pypa/pip/issues/12095></code>_)</li> <li>Fix completion script for zsh (<code>[#12166](pypa/pip#12166) <https://github.com/pypa/pip/issues/12166></code>_)</li> <li>Fix improper handling of the new onexc argument of <code>shutil.rmtree()</code> in Python 3.12. (<code>[#12187](pypa/pip#12187) <https://github.com/pypa/pip/issues/12187></code>_)</li> <li>Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (<code>[#12225](pypa/pip#12225) <https://github.com/pypa/pip/issues/12225></code>_)</li> <li>Fix crash when the git version number contains something else than digits and dots. (<code>[#12280](pypa/pip#12280) <https://github.com/pypa/pip/issues/12280></code>_)</li> <li>Use <code>-r=...</code> instead of <code>-r ...</code> to specify references with Mercurial. (<code>[#12306](pypa/pip#12306) <https://github.com/pypa/pip/issues/12306></code>_)</li> <li>Redact password from URLs in some additional places. (<code>[#12350](pypa/pip#12350) <https://github.com/pypa/pip/issues/12350></code>_)</li> <li>pip uses less memory when caching large packages. As a result, there is a new on-disk cache format stored in a new directory ($PIP_CACHE_DIR/http-v2). (<code>[#2984](pypa/pip#2984) <https://github.com/pypa/pip/issues/2984></code>_)</li> </ul> <h2>Vendored Libraries</h2> <ul> <li>Upgrade certifi to 2023.7.22</li> <li>Add truststore 0.8.0</li> <li>Upgrade urllib3 to 1.26.17</li> </ul> <p>Improved Documentation</p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/e3dc91dad93a020b3034a87ebe59027f63370fe8"><code>e3dc91d</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/3e85558b10722598fb3353126e2f19979f7cf7dd"><code>3e85558</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/8d0278771c7325b04f02cb073c8ef02827cbeb93"><code>8d02787</code></a> Reclassify news fragment</li> <li><a href="https://github.com/pypa/pip/commit/f6ecf406c3929b3127ddb480ef4350542d102338"><code>f6ecf40</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12350">#12350</a> from sbidoul/readact-collecting-url</li> <li><a href="https://github.com/pypa/pip/commit/306086513bd1a6500126057492ee8b0f9a2e79dd"><code>3060865</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12335">#12335</a> from edmorley/patch-1</li> <li><a href="https://github.com/pypa/pip/commit/8f0ed32413daa411a728b50cd7776b9c02b010d5"><code>8f0ed32</code></a> Redact URLs in Collecting... logs</li> <li><a href="https://github.com/pypa/pip/commit/d1659b87e46abd0a2dcc74f2160dd52e6190e13b"><code>d1659b8</code></a> Correct issue number for NEWS entry added by <a href="https://github.com/pypa/pip/issues/12197">#12197</a></li> <li><a href="https://github.com/pypa/pip/commit/2333ef3b53a71fb7acc9e76d6ff90409576b2250"><code>2333ef3</code></a> Upgrade urllib3 to 1.26.17 (<a href="https://github.com/pypa/pip/issues/12343">#12343</a>)</li> <li><a href="https://github.com/pypa/pip/commit/496b268c1b9ce3466c08eb4819e5460a943d1793"><code>496b268</code></a> Update "Running Tests" documentation (<a href="https://github.com/pypa/pip/issues/12334">#12334</a>)</li> <li><a href="https://github.com/pypa/pip/commit/d1f0981cb2af3c72ff871b54a8a98581ccb2890a"><code>d1f0981</code></a> Merge pull request <a href="https://github.com/pypa/pip/issues/12331">#12331</a> from sbidoul/update-egg-deprecation-message</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.3.1...23.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.3.1&new-version=23.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

sanderr added 8 commits June 20, 2023 17:13

take non-extra requirements into account for extra installs

5bebe85

partial improvement

937d8f0

refinements

5f8f40e

fixes

d09431f

cleanup

49027d7

reverted troublesome changes

cb0f97f

improvement

3160293

stray todo

1038f15

sanderr marked this pull request as ready for review June 26, 2023 09:50

sanderr mentioned this pull request Jun 28, 2023

Investigate incorporating extras merging into the resolver sarugaku/resolvelib#14

Open

uranusjr reviewed Jul 6, 2023

View reviewed changes

src/pip/_internal/resolution/resolvelib/requirements.py Outdated Show resolved Hide resolved

uranusjr reviewed Jul 6, 2023

View reviewed changes

sanderr added 3 commits July 13, 2023 14:57

dropped unused attribute

8aa1758

use regex for requirement update

faa3289

clarification

7e8da61

sanderr requested a review from uranusjr July 13, 2023 15:03

sanderr added 6 commits July 25, 2023 15:36

added resolver test case

ff9aeae

added test for comes-from reporting

3fa373c

added test case for report bugfix

e569017

added second report test case

cc6a2bd

Merge branch 'main' into issue/11924-requirements-on-extras

fc86308

news entries

4ae829c

psf-chronographer bot added the bot:chronographer:provided label Jul 25, 2023

sanderr commented Jul 25, 2023

View reviewed changes

news/12095.bugfix.rst Show resolved Hide resolved

sanderr added 3 commits July 25, 2023 17:19

py38 compatibility

dc01a40

py37 compatibility

292387f

fixed minor type errors

39e1102

uranusjr reviewed Jul 26, 2023

View reviewed changes

sanderr added 2 commits July 26, 2023 10:32

linting

e6333bb

made primary news fragment of type feature

1207389

pypa-bot added the needs rebase or merge PR has conflicts with current master label Sep 13, 2023

Merge branch 'main' into issue/11924-requirements-on-extras

89b68c6

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 13, 2023

made assertions more robust

0f543e3

sanderr requested a review from pradyunsg September 13, 2023 14:53

pradyunsg approved these changes Sep 17, 2023

View reviewed changes

sanderr requested a review from uranusjr September 18, 2023 08:15

pradyunsg added this to the 23.3 milestone Oct 4, 2023

uranusjr approved these changes Oct 5, 2023

View reviewed changes

uranusjr merged commit b551c09 into pypa:main Oct 5, 2023
24 checks passed

sanderr deleted the issue/11924-requirements-on-extras branch October 20, 2023 13:12

AndydeCleyre mentioned this pull request Oct 26, 2023

latest pip (23.3) causes dependency oddity jazzband/pip-tools#2003

Closed

edmorley mentioned this pull request Oct 26, 2023

Pip install local with extra fails dependency resolution #12372

Closed

1 task

sanderr mentioned this pull request Oct 30, 2023

Pip fails to install some satisfiable requirements since extras optimization #12376

Closed

1 task

multimeric mentioned this pull request Oct 30, 2023

Fix pip related requirements issue BioimageAnalysisCoreWEHI/napari_lattice#55

Open

notatallshaw added a commit to notatallshaw/pip that referenced this pull request Nov 1, 2023

Revert "Merge pull request pypa#12095 from sanderr/issue/11924-requir…

b585342

…ements-on-extras" This reverts commit b551c09, reversing changes made to dfaac0a.

github-actions bot locked as resolved and limited conversation to collaborators Nov 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve handling of constraints on requirements with extras #12095

Improve handling of constraints on requirements with extras #12095

sanderr commented Jun 22, 2023 •

edited

Loading

uranusjr Jul 6, 2023 •

edited

Loading

sanderr Jul 13, 2023

pfmoore Jul 13, 2023

pradyunsg Jul 25, 2023

sanderr Jul 25, 2023

uranusjr Jul 26, 2023

sanderr Jul 26, 2023

uranusjr Jul 26, 2023

sanderr Jul 26, 2023

sanderr commented Oct 4, 2023 •

edited

Loading

pradyunsg commented Oct 4, 2023

uranusjr left a comment

uranusjr commented Oct 5, 2023

AndydeCleyre commented Oct 26, 2023

sanderr commented Oct 26, 2023 •

edited

Loading

Improve handling of constraints on requirements with extras #12095

Improve handling of constraints on requirements with extras #12095

Conversation

sanderr commented Jun 22, 2023 • edited Loading

Description

Status

Examples

uranusjr Jul 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderr commented Oct 4, 2023 • edited Loading

pradyunsg commented Oct 4, 2023

uranusjr left a comment

Choose a reason for hiding this comment

uranusjr commented Oct 5, 2023

AndydeCleyre commented Oct 26, 2023

sanderr commented Oct 26, 2023 • edited Loading

sanderr commented Jun 22, 2023 •

edited

Loading

uranusjr Jul 6, 2023 •

edited

Loading

sanderr commented Oct 4, 2023 •

edited

Loading

sanderr commented Oct 26, 2023 •

edited

Loading